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We present an update of the search for the lepton family number violating 
decay r — > fi'y using 12.6 million t + t~ pairs collected with the CLEO detector. 
No evidence of a signal has been found and the corresponding upper limit is 

\^0 . B(r — ► fi^f) < 1.1 x 10 -6 at 90% CL, significantly smaller than previous 

^D | experimental limits. 
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Non-conservation of the lepton flavor is expected in many extensions of the standard 
model and searches for lepton flavor violating decays provide strong constraints on possible 
new physics processes. Although there are many possible r decay channels which do not 
conserve the lepton flavor number, the decay r — > /17 is favored by most theoretical extensions 
of the Standard Model JTJ . The most optimistic predictions for rates of such decays are based 
on the supersymmetric models 0-Q , on the left-right supersymmetric models || and on the 
super symmetric string unified models ||. Recent calculations [f|,|| predict values for the 
branching fraction of the decay r — > /17 at the order of a few times 10 -6 for some ranges of 
model parameters. In general, the expectations for all other lepton number or lepton flavor 
violating decays of the r are at least an order of magnitude lower. Experimental searches for 
the r — > /ry decay are limited by the number of observed r decays. The lowest upper limit @ 
of £>(r — > fij) < 3.0 x 10~ 6 at 90% CL has been published by the CLEO Collaboration using 
4.24 million t + t~ pairs. The results presented here supersede the results of the previous 
CLEO analysis 0. 

In this analysis we use a data sample from the reaction e + e~ — > t + t~ collected at CESR 
at or near the energy of the T(4S). The data correspond to a total integrated luminosity 
of 13.8 fb _1 and contain 12.6 million t + t~ pairs. The CLEO detectors employed here are 
described in Refs. ||J§. The event selection follows the procedure used in the previous 
search [7[|. We select events with a 1-vs-l topology, where the signal candidate r decays into 
/ry and the tag side includes all standard r decays into one charged particle, any number of 
photons and at least one neutrino. 

We select t + t~ pair events with exactly two good charged tracks, with total charge 
equal to zero, and with the angle between the charged tracks greater than 90°. Because 
radiative /i-pair production produces high background rates, we allow only one identified 
muon per event. In addition, each candidate event must have exactly one photon separated 
by more than 20° from the closest charged track projection onto the calorimeter in the muon 
hemisphere. This photon must lie in the calorimeter barrel (i.e., | cos# 7 | < 0.71, where 9 1 
is an angle between the photon and beam direction), have a photon-like lateral profile and 
have energy deposition in the calorimeter greater than 300 MeV. This minimum energy cut 
is dictated by the kinematics of a 2-body r decay. The angle between the direction of the 
photon and the momentum of the muon track must satisfy 0.4 < cos# M7 < 0.8, where the 
upper limit is again dictated by kinematics, and the lower limit is obtained by optimizing 
the signal-to-background ratio. 

The main sources of background in the selected samples are due to /i-pair production, 
radiative r — > [x^vv decays, and two-photon processes. To minimize these backgrounds, 
we require that the cosine of the angle between the total missing momentum of the event 
and the momentum of the tagging particle be greater than 0.4. The missing momentum is 
calculated as the negative of the sum of momenta of the two charged tracks and all neutral 
showers detected in the calorimeter with energies above 30 MeV. Because there must be at 
least one undetected neutrino on the tag side, the missing momentum in an event having 
t — ► ^7 is expected to fall into the tagging track hemisphere, while for all radiative processes 
the missing momentum should be uncorrelated with the charged track on the tag side. The 
neutrino emission on the tag side should also result in a large total transverse momentum 
with respect to the beam direction. Thus, to suppress background produced by copious 
two-photon and radiative QED processes, we require that the total transverse momentum of 



the event be greater than 300 MeV/c. The selection efficiency of all requirements above is 
estimated from Monte Carlo simulation as 16.2%. 

Final signal selection criteria are based on kinematic constraints since a neutrinoless r 
decay should have a total energy and an effective mass of the /ry consistent with the beam 
energy and r mass, respectively. To determine these final criteria, we employ two different 
techniques. First, we follow the method outlined in CLEO's previous search [0] for the 
decay r — > /ry. Then we perform a more sensitive analysis based on an unbinned extended 
maximum likelihood (EML) fit to the data. 

Following the method described in detail in Ref. fTJ, we parameterize the signal Monte 
Carlo mass and energy distributions separately as tailed Gaussian densities. Initial and 
final state radiation produces an asymmetric tail in energy, and both mass and energy 
distributions are slightly distorted by an asymmetric response of the calorimeter. The energy 
density is given by 

f f E ) = { { 1/ [v(~E + l/v - V)} } l exp(-^/2) ; E < -q ; (1) 

I exp(-E 2 /2); E>- V ; 

where E = (E — Ef, eam ) / a e and cr^, 77, and I are the fit parameters. A similar formula is used 
for the invariant mass of the /ry system, fh = (m — m T )/a m . The r mass, m T , is taken to be 
1.777 GeV/c 2 [ID], and the beam energy Eb eam varies from 5.26 to 5.29 GeV. The obtained 
Gaussian resolutions are a m = 23.2 ± 0.4 MeV/c 2 and a E = 47.9 ± 1.2 MeV. The signal 
region is then defined to be within ±3 standard deviations of the fitted Gaussian component 
of the distribution. There are 6 events observed in the signal region shown as the central 
box in Fig. [I]. To estimate the amount of background expected in the signal region, we 
extrapolate the data from the sideband. We assume that the background distributions are 
linear in the vicinity of m T and Ef, eam and define the sideband regions to be between 5 and 8 
standard deviations as shown in Fig. [I]. To estimate the background uncertainty associated 
with this technique, we vary the sideband definition. The total expected background in the 
signal region is estimated as 5.5 ± 0.5 events. 

The upper limit on the r — > /ry branching fraction is estimated following the Bayesian 



prescription [11.12 



e- b E n nUb n /n\ Ui ' [A) 

where So is an upper limit on the number of events in the signal region at 90% CL, b is the 
expected background rate, and n Q is the number of observed events. The upper limit on the 
branching fraction is then 

B(r - // 7 ) < ^~ at 90% CL , (3) 

where e is the event selection efficiency and N TT is the total number of r-pairs produced. 
Applying this technique, we obtain an upper limit on the branching fraction 13 (r — * yrf) of 
1.8 x 10~ 6 at 90% CL. 
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FIG. 1. (Eur, - E beam ) vs (m 



in 



m T ) distribution. The data are shown with solid squares 
and the signal Monte Carlo distribution is shown with open circles. The central box represents the 
signal region and the four other boxes represent the sidebands. 



The systematic uncertainty in detector sensitivity 2eN TT is conservatively estimated as 
10%. This uncertainty is obtained by adding in quadrature uncertainties in track recon- 
struction efficiency (3%), photon reconstruction efficiency (5%), cut selection (5%), luminos- 
ity and cross-section (1.4%), lepton identification (4%), Monte Carlo statistics (1.5%) and 
trigger efficiency (5%). The upper limit for the branching fraction is also affected by the 
uncertainty in the background estimate of 0.5 events. To incorporate systematic uncertainty 
into the upper limit, we assume that the errors related to 2eN TT and to the background 
estimate have Gaussian distributions and apply a technique described in Refs. [0,0] ■ This 
technique reweights the probability (||) by a Gaussian probability density of the detector 
sensitivity 2eN TT and a Gaussian probability density of the number of background events 
b. The incorporation of these systematic uncertainties increases the upper limit by 1.9% of 
itself. 

A more sensitive upper limit is obtained by performing an unbinned EML fit which takes 
into account the details of the distributions and correlations between the mass and energy 
of signal event candidates. The likelihood function is defined as 



C(s,b) 



-(s+b) N 



N\ 



liisSi + bBi) 



(4) 



s=l 



where N is the number of events in the signal region and its vicinity, s and b are the numbers 
of signal and background events, respectively, and Si and Bi are the signal and background 
densities, respectively. The signal distribution is described by a two-dimensional Gaussian 
and a non-Gaussian tail in energy produced by initial and final state radiation. This tail 
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covers the region below the beam energy and is modeled by a gamma-function. 

— — (m 2 - 2pmE + E 

2(1 -p 2 V ^ 



£k(m,E)= J^^ exp 



2na m a E \/l - p 1 



+ A T ((m,E) ; 

(5) 

I i ™^ i ^,1 /o\ i ( f?\ a ~,^ ( t? i o\ \t t? ^ n . 
C(m, E) = < V^^ 



^- exp (-m 2 /2) ^J^ (-E) - 1 exp (£//?) if E < 



otherwise ; 

where A G and A^ are the relative contributions of the Gaussian component and the non- 
Gaussian tail with the sum of Ac + At constrained to unity, o m and oe are mass and energy 
resolutions, respectively, p is the correlation coefficient, and a and (3 define the shape of 
the non-Gaussian tail ((m,E). To obtain the parameters of the signal density Si, we fit 
the signal Monte Carlo distribution. The extracted value of the correlation coefficient is 
p = 0.625 ±0.012, the relative areas A G and A T are 0.81 ±0.02 and 0.19 ±0.02, respectively, 
and the resolutions a m and oe are close to those obtained in the one- dimensional fits (jl]). 
The background is parameterized by a function linear in energy with the coefficients ao and 
a\ obtained from a fit to the data: 

5i(m ' E) = m 2 - rm (ao - a lEbeam )(E 2 - k) ± 0.5a l( £| - Ef) [ao + ^ " Eb ^ )] ' (6) 

where {mi,m2) and (Ei,E 2 ) are the limits defining the fit region. The region within 4 
standard deviations near the beam energy Eb eam is excluded from the fit to avoid bias caused 
by the possible presence of real signal events in this region. Uncertainties of the background 
shape parameters a and g^ are estimated by varying the number of bins in the fit region. 

The EML fit to the data gives the number of candidates for the decay r — *■ yq as 1.8 
events with an estimated statistical significance of the signal 1.0 standard deviations. The 
fit region, shown in Fig. [I], is defined to be within 10 standard deviations near the r mass 
and beam energy. The total number of events in the fit region is 53. 

To estimate the upper limit, we use a method [14] developed for unbinned EML fits. Q 



The expected number of background events is fixed at the value extracted from the EML 
fit to the data. For every assumed expected number of signal events s, we generate 10,000 
Monte Carlo samples. For every sample, we generate numbers of signal and background 
events using Poisson distributions and then we generate positions of these events on the 
energy- vs-mass plane using the densities from Eqns. (|5|) and (|6|). For each sample we then 
perform an unbinned EML fit to extract the number of signal events, following the same 
procedure as for the data. The confidence level corresponding to this value of s is defined 
as a fraction of samples where the extracted number of events exceeds that observed in the 
data, i.e., 1.8. We repeat this procedure until we find a value of s = s that gives a 90% CL. 



1 This method assumes a confidence interval to be of the form (0, so) and thus gives a different 



upper limit than that obtained by the method of Ref. [15]. The prescription |lj| has been developed 
for problems with integer numbers of observed signal candidate events and, in its present shape, is 
inapplicable to EML fits. 
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This value has to be divided by the selection efficiency and the number of produced r-pairs 
in accordance with Eqn. (|3]). The obtained upper limit on the branching fraction £>(r — > fi'-f) 
is 1.0 x 10" 6 at 90% CL. 

To incorporate systematic uncertainty into this result, we smear the background shape 
parameters a^ and a x within the estimated errors assuming Gaussian distributions and taking 
into account the correlation between these two parameters. We then repeat the procedure 
described in the previous paragraph integrating the likelihood function over the parameter 
space of a,Q and a±. We do not observe a significant signal contribution, and the parameters 
of the signal density are known with high accuracy; thus, the effect of uncertainties in these 
parameters is negligible. In addition to smearing the background shape, we integrate the 
quantity l/(2eN TT ) assuming a Gaussian distribution for the detector sensitivity 2eN TT with 
a relative standard deviation equal to the estimated systematic uncertainty of 10%. The 
incorporation of these systematic uncertainties increases the upper limit by 13% of itself. 
This uncertainty is dominated by the errors in the background shape parameters. 

The selection efficiencies, numbers of events, and upper limits calculated with and with- 
out inclusion of systematic errors for both techniques are given in Table |. This result is 
limited by the total integrated luminosity and represents a significant improvement over the 
previous analysis 0]. The obtained upper limit of 1.1 x 10 -6 restricts the parameter space 
of models |,§. 

TABLE I. 
Selection efficiencies, numbers of events, and upper limits calculated with and without sys- 
tematic errors. 







Method of Ref. 


IZI 


Unbinned EML fit 


MC efficiency, e 




12.7% 




15.2% 


Number of signal events 




n = 6 




8 = 1.8 


Expected background rate, b 




5.5 ±0.5 




- 


Statistical significance of the signal 




- 




1.0a 


Upper limit at 90% CL, so 




5.8 




3.8 


Upper limit for B(t -► // 7 ) at 90% CL 




1.8 x 10~ 6 




1.0 x 10~ 6 


Upper limit at 90% CL with systematic error 


included 


1.8 x 10" 6 




1.1 x 10~ 6 
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